---
layout: post
title: "Testing implementations of LibFM"
date: "2016-02-15"
author: Alex Rogozhnikov
tags:
- notebook
---

<style>
// proper alignment of cells and smaller numbers
table.dataframe td {
  text-align: right;
  font-size: 0.9em;
}
</style>

<div class="cell border-box-sizing text_cell rendered">
<div class="prompt input_prompt">
</div>
<div class="inner_cell">
<div class="text_cell_render border-box-sizing rendered_html">
<h2 id="Testing-implementations-of-LibFM">Testing implementations of LibFM<a class="anchor-link" href="#Testing-implementations-of-LibFM">&#182;</a></h2><p>By <strong>LibFM</strong> I mean an approach to solve classification and regression problems.
This approach is frequently used in recommendation systems, because it generalizes the matrix decompositions.</p>
<p><strong>LibFM</strong> proved to be quite useful to deal with highly categorical data (user id / movie id / movie language / site id / advertisement id / etc.).</p>
<p>Implementations tested</p>
<ul>
<li>Original and widely known implementation was written by Steffen Rendle (and available on <a href="https://github.com/srendle/libfm">github</a>).<ul>
<li>contains SGD, SGDA, ALS and MCMC optimizers</li>
<li>command-line interface, does not have official python / R wrapper</li>
<li>does not provide a way to save / load trained formula. Each time you want to predict something, you need to restart training process </li>
<li>has some extensions (that almost nobody uses)</li>
<li>supports linux, mac os</li>
<li>has non-oficial <a href="https://github.com/jfloff/pywFM">pythonic wrapper</a></li>
</ul>
</li>
</ul>
<ul>
<li>FastFM (<a href="https://github.com/ibayer/fastFM">github repo</a>)<ul>
<li>claimed to be faster in the author's article</li>
<li>has both command-line interface and convenient python wrapper, which <em>almost</em> follows scikit-learn conventions.</li>
<li>supports SGD, ALS and MCMC optimizers</li>
<li>supports save / load (for the except of MCMC)</li>
<li>supports linux, mac os (though some issues with mac os)</li>
</ul>
</li>
</ul>
<ul>
<li>pylibFM (<a href="https://github.com/coreylynch/pyFM">github repo</a>)<ul>
<li>uses SGDA</li>
<li>pythonic library implemented with cython</li>
<li>save / load operates normally</li>
<li>supports any platform, provided cython operates normally</li>
<li>slow and requires additional tuning, the number of iterations is reduced for pylibFM in tests</li>
</ul>
</li>
</ul>
<p>None of the libraries are pip-installable and all libraries need some manual setup. FastFM is the only to install itself normally into site-packages.</p>

</div>
</div>
</div>
<div class="cell border-box-sizing text_cell rendered">
<div class="prompt input_prompt">
</div>
<div class="inner_cell">
<div class="text_cell_render border-box-sizing rendered_html">
<h2 id="What-is-tested">What is tested<a class="anchor-link" href="#What-is-tested">&#182;</a></h2><p>ALS (alternating least squares) is very useful optimization technique for factorization models, however
there is still one parameter one has to pass - namely, regularization. Quality of classification / regression is quite sensible to this parameter, so for fast tests data analyst prefers to leave the question of selecting regularization to machine learning.</p>
<p>MCMC is usually proposed as a solution: optimization algorithm should "find" the optimal regularization. 
MCMC uses however some priors (which don't influence the result that much).</p>
<p>So I am testing the quality libraries provide <strong>without additional tuning</strong> to check how bayesian inference and other heuristics work.</p>
<h2 id="Logistic-regression">Logistic regression<a class="anchor-link" href="#Logistic-regression">&#182;</a></h2><p>Logistic regression is used as a stable <strong>baseline</strong>, because it is basic method to work with highly categorical data.</p>
<p>However, logistic regression, for instance, does not encounter the relation between user variables and movie variables (in the context of movie recommendations), so this approach is not able to provide any senseful recommendations.</p>

</div>
</div>
</div>
<div class="cell border-box-sizing code_cell rendered">
<div class="input">
<div class="prompt input_prompt">In&nbsp;[1]:</div>
<div class="inner_cell">
    <div class="input_area">
<div class=" highlight hl-ipython2"><pre><span class="kn">import</span> <span class="nn">numpy</span>
<span class="kn">import</span> <span class="nn">pandas</span>
<span class="kn">import</span> <span class="nn">load_problems</span>
<span class="kn">import</span> <span class="nn">cPickle</span> <span class="kn">as</span> <span class="nn">pickle</span>
<span class="kn">from</span> <span class="nn">sklearn.metrics</span> <span class="kn">import</span> <span class="n">roc_auc_score</span><span class="p">,</span> <span class="n">mean_squared_error</span>
</pre></div>

</div>
</div>
</div>

</div>
<div class="cell border-box-sizing code_cell rendered">
<div class="input">
<div class="prompt input_prompt">In&nbsp;[2]:</div>
<div class="inner_cell">
    <div class="input_area">
<div class=" highlight hl-ipython2"><pre><span class="kn">from</span> <span class="nn">fastFM.mcmc</span> <span class="kn">import</span> <span class="n">FMClassification</span><span class="p">,</span> <span class="n">FMRegression</span>
<span class="kn">from</span> <span class="nn">sklearn.preprocessing</span> <span class="kn">import</span> <span class="n">OneHotEncoder</span>
<span class="kn">from</span> <span class="nn">sklearn.linear_model</span> <span class="kn">import</span> <span class="n">LogisticRegression</span><span class="p">,</span> <span class="n">Ridge</span>
<span class="kn">from</span> <span class="nn">sklearn.datasets</span> <span class="kn">import</span> <span class="n">dump_svmlight_file</span>
</pre></div>

</div>
</div>
</div>

</div>
<div class="cell border-box-sizing text_cell rendered">
<div class="prompt input_prompt">
</div>
<div class="inner_cell">
<div class="text_cell_render border-box-sizing rendered_html">
<h2 id="Defining-functions-for-benchmarking">Defining functions for benchmarking<a class="anchor-link" href="#Defining-functions-for-benchmarking">&#182;</a></h2>
</div>
</div>
</div>
<div class="cell border-box-sizing code_cell rendered">
<div class="input">
<div class="prompt input_prompt">In&nbsp;[3]:</div>
<div class="inner_cell">
    <div class="input_area">
<div class=" highlight hl-ipython2"><pre><span class="n">LIBFM_PATH</span> <span class="o">=</span> <span class="s">&#39;/moosefs/ipython_env/python_libfm/bin/libFM&#39;</span>
<span class="n">PYLIBFM_PATH</span> <span class="o">=</span> <span class="s">&#39;/moosefs/ipython_env/python_pylibFM/&#39;</span>

<span class="kn">import</span> <span class="nn">sys</span>
<span class="k">if</span> <span class="n">PYLIBFM_PATH</span> <span class="ow">not</span> <span class="ow">in</span> <span class="n">sys</span><span class="o">.</span><span class="n">path</span><span class="p">:</span>
    <span class="n">sys</span><span class="o">.</span><span class="n">path</span><span class="o">.</span><span class="n">insert</span><span class="p">(</span><span class="mi">0</span><span class="p">,</span> <span class="n">PYLIBFM_PATH</span><span class="p">)</span>
<span class="kn">import</span> <span class="nn">pylibfm</span>


<span class="k">def</span> <span class="nf">fitpredict_logistic</span><span class="p">(</span><span class="n">trainX</span><span class="p">,</span> <span class="n">trainY</span><span class="p">,</span> <span class="n">testX</span><span class="p">,</span> <span class="n">classification</span><span class="o">=</span><span class="bp">True</span><span class="p">,</span> <span class="o">**</span><span class="n">params</span><span class="p">):</span>
    <span class="n">encoder</span> <span class="o">=</span> <span class="n">OneHotEncoder</span><span class="p">(</span><span class="n">handle_unknown</span><span class="o">=</span><span class="s">&#39;ignore&#39;</span><span class="p">)</span><span class="o">.</span><span class="n">fit</span><span class="p">(</span><span class="n">trainX</span><span class="p">)</span>
    <span class="n">trainX</span> <span class="o">=</span> <span class="n">encoder</span><span class="o">.</span><span class="n">transform</span><span class="p">(</span><span class="n">trainX</span><span class="p">)</span>
    <span class="n">testX</span> <span class="o">=</span> <span class="n">encoder</span><span class="o">.</span><span class="n">transform</span><span class="p">(</span><span class="n">testX</span><span class="p">)</span>
    <span class="k">if</span> <span class="n">classification</span><span class="p">:</span>
        <span class="n">clf</span> <span class="o">=</span> <span class="n">LogisticRegression</span><span class="p">(</span><span class="o">**</span><span class="n">params</span><span class="p">)</span>
        <span class="n">clf</span><span class="o">.</span><span class="n">fit</span><span class="p">(</span><span class="n">trainX</span><span class="p">,</span> <span class="n">trainY</span><span class="p">)</span>
        <span class="k">return</span> <span class="n">clf</span><span class="o">.</span><span class="n">predict_proba</span><span class="p">(</span><span class="n">testX</span><span class="p">)[:,</span> <span class="mi">1</span><span class="p">]</span>
    <span class="k">else</span><span class="p">:</span>
        <span class="n">clf</span> <span class="o">=</span> <span class="n">Ridge</span><span class="p">(</span><span class="o">**</span><span class="n">params</span><span class="p">)</span>
        <span class="n">clf</span><span class="o">.</span><span class="n">fit</span><span class="p">(</span><span class="n">trainX</span><span class="p">,</span> <span class="n">trainY</span><span class="p">)</span>
        <span class="k">return</span> <span class="n">clf</span><span class="o">.</span><span class="n">predict</span><span class="p">(</span><span class="n">testX</span><span class="p">)</span>

<span class="k">def</span> <span class="nf">fitpredict_fastfm</span><span class="p">(</span><span class="n">trainX</span><span class="p">,</span> <span class="n">trainY</span><span class="p">,</span> <span class="n">testX</span><span class="p">,</span> <span class="n">classification</span><span class="o">=</span><span class="bp">True</span><span class="p">,</span> <span class="n">rank</span><span class="o">=</span><span class="mi">8</span><span class="p">,</span> <span class="n">n_iter</span><span class="o">=</span><span class="mi">100</span><span class="p">):</span>
    <span class="n">encoder</span> <span class="o">=</span> <span class="n">OneHotEncoder</span><span class="p">(</span><span class="n">handle_unknown</span><span class="o">=</span><span class="s">&#39;ignore&#39;</span><span class="p">)</span><span class="o">.</span><span class="n">fit</span><span class="p">(</span><span class="n">trainX</span><span class="p">)</span>
    <span class="n">trainX</span> <span class="o">=</span> <span class="n">encoder</span><span class="o">.</span><span class="n">transform</span><span class="p">(</span><span class="n">trainX</span><span class="p">)</span>
    <span class="n">testX</span> <span class="o">=</span> <span class="n">encoder</span><span class="o">.</span><span class="n">transform</span><span class="p">(</span><span class="n">testX</span><span class="p">)</span>
    <span class="k">if</span> <span class="n">classification</span><span class="p">:</span>
        <span class="n">clf</span> <span class="o">=</span> <span class="n">FMClassification</span><span class="p">(</span><span class="n">rank</span><span class="o">=</span><span class="n">rank</span><span class="p">,</span> <span class="n">n_iter</span><span class="o">=</span><span class="n">n_iter</span><span class="p">)</span>
        <span class="k">return</span> <span class="n">clf</span><span class="o">.</span><span class="n">fit_predict_proba</span><span class="p">(</span><span class="n">trainX</span><span class="p">,</span> <span class="n">trainY</span><span class="p">,</span> <span class="n">testX</span><span class="p">)</span>
    <span class="k">else</span><span class="p">:</span>
        <span class="n">clf</span> <span class="o">=</span> <span class="n">FMRegression</span><span class="p">(</span><span class="n">rank</span><span class="o">=</span><span class="n">rank</span><span class="p">,</span> <span class="n">n_iter</span><span class="o">=</span><span class="n">n_iter</span><span class="p">)</span>
        <span class="k">return</span> <span class="n">clf</span><span class="o">.</span><span class="n">fit_predict</span><span class="p">(</span><span class="n">trainX</span><span class="p">,</span> <span class="n">trainY</span><span class="p">,</span> <span class="n">testX</span><span class="p">)</span>  

<span class="k">def</span> <span class="nf">fitpredict_libfm</span><span class="p">(</span><span class="n">trainX</span><span class="p">,</span> <span class="n">trainY</span><span class="p">,</span> <span class="n">testX</span><span class="p">,</span> <span class="n">classification</span><span class="o">=</span><span class="bp">True</span><span class="p">,</span> <span class="n">rank</span><span class="o">=</span><span class="mi">8</span><span class="p">,</span> <span class="n">n_iter</span><span class="o">=</span><span class="mi">100</span><span class="p">):</span>
    <span class="n">encoder</span> <span class="o">=</span> <span class="n">OneHotEncoder</span><span class="p">(</span><span class="n">handle_unknown</span><span class="o">=</span><span class="s">&#39;ignore&#39;</span><span class="p">)</span><span class="o">.</span><span class="n">fit</span><span class="p">(</span><span class="n">trainX</span><span class="p">)</span>
    <span class="n">trainX</span> <span class="o">=</span> <span class="n">encoder</span><span class="o">.</span><span class="n">transform</span><span class="p">(</span><span class="n">trainX</span><span class="p">)</span>
    <span class="n">testX</span> <span class="o">=</span> <span class="n">encoder</span><span class="o">.</span><span class="n">transform</span><span class="p">(</span><span class="n">testX</span><span class="p">)</span>
    <span class="n">train_file</span> <span class="o">=</span> <span class="s">&#39;libfm_train.txt&#39;</span>
    <span class="n">test_file</span> <span class="o">=</span> <span class="s">&#39;libfm_test.txt&#39;</span>
    <span class="k">with</span> <span class="nb">open</span><span class="p">(</span><span class="n">train_file</span><span class="p">,</span> <span class="s">&#39;w&#39;</span><span class="p">)</span> <span class="k">as</span> <span class="n">f</span><span class="p">:</span>
        <span class="n">dump_svmlight_file</span><span class="p">(</span><span class="n">trainX</span><span class="p">,</span> <span class="n">trainY</span><span class="p">,</span> <span class="n">f</span><span class="o">=</span><span class="n">f</span><span class="p">)</span>
    <span class="k">with</span> <span class="nb">open</span><span class="p">(</span><span class="n">test_file</span><span class="p">,</span> <span class="s">&#39;w&#39;</span><span class="p">)</span> <span class="k">as</span> <span class="n">f</span><span class="p">:</span>
        <span class="n">dump_svmlight_file</span><span class="p">(</span><span class="n">testX</span><span class="p">,</span> <span class="n">numpy</span><span class="o">.</span><span class="n">zeros</span><span class="p">(</span><span class="n">testX</span><span class="o">.</span><span class="n">shape</span><span class="p">[</span><span class="mi">0</span><span class="p">]),</span> <span class="n">f</span><span class="o">=</span><span class="n">f</span><span class="p">)</span>
    <span class="n">task</span> <span class="o">=</span> <span class="s">&#39;c&#39;</span> <span class="k">if</span> <span class="n">classification</span> <span class="k">else</span> <span class="s">&#39;r&#39;</span>
    <span class="n">console_output</span> <span class="o">=</span> <span class="o">!</span><span class="nv">$LIBFM_PATH</span> -task <span class="nv">$task</span> -method mcmc -train <span class="nv">$train_file</span> -test <span class="nv">$test_file</span> -iter <span class="nv">$n_iter</span> -dim <span class="s1">&#39;1,1,$rank&#39;</span> -out output.libfm
    
    <span class="n">libfm_pred</span> <span class="o">=</span> <span class="n">pandas</span><span class="o">.</span><span class="n">read_csv</span><span class="p">(</span><span class="s">&#39;output.libfm&#39;</span><span class="p">,</span> <span class="n">header</span><span class="o">=</span><span class="bp">None</span><span class="p">)</span><span class="o">.</span><span class="n">values</span><span class="o">.</span><span class="n">flatten</span><span class="p">()</span>
    <span class="k">return</span> <span class="n">libfm_pred</span>

<span class="k">def</span> <span class="nf">fitpredict_pylibfm</span><span class="p">(</span><span class="n">trainX</span><span class="p">,</span> <span class="n">trainY</span><span class="p">,</span> <span class="n">testX</span><span class="p">,</span> <span class="n">classification</span><span class="o">=</span><span class="bp">True</span><span class="p">,</span> <span class="n">rank</span><span class="o">=</span><span class="mi">8</span><span class="p">,</span> <span class="n">n_iter</span><span class="o">=</span><span class="mi">10</span><span class="p">):</span>
    <span class="n">encoder</span> <span class="o">=</span> <span class="n">OneHotEncoder</span><span class="p">(</span><span class="n">handle_unknown</span><span class="o">=</span><span class="s">&#39;ignore&#39;</span><span class="p">)</span><span class="o">.</span><span class="n">fit</span><span class="p">(</span><span class="n">trainX</span><span class="p">)</span>
    <span class="n">trainX</span> <span class="o">=</span> <span class="n">encoder</span><span class="o">.</span><span class="n">transform</span><span class="p">(</span><span class="n">trainX</span><span class="p">)</span>
    <span class="n">testX</span> <span class="o">=</span> <span class="n">encoder</span><span class="o">.</span><span class="n">transform</span><span class="p">(</span><span class="n">testX</span><span class="p">)</span>
    <span class="n">task</span> <span class="o">=</span> <span class="s">&#39;classification&#39;</span> <span class="k">if</span> <span class="n">classification</span> <span class="k">else</span> <span class="s">&#39;regression&#39;</span>
    <span class="n">fm</span> <span class="o">=</span> <span class="n">pylibfm</span><span class="o">.</span><span class="n">FM</span><span class="p">(</span><span class="n">num_factors</span><span class="o">=</span><span class="n">rank</span><span class="p">,</span> <span class="n">num_iter</span><span class="o">=</span><span class="n">n_iter</span><span class="p">,</span> <span class="n">verbose</span><span class="o">=</span><span class="bp">False</span><span class="p">,</span> <span class="n">task</span><span class="o">=</span><span class="n">task</span><span class="p">)</span>
    <span class="k">if</span> <span class="n">classification</span><span class="p">:</span>
        <span class="n">fm</span><span class="o">.</span><span class="n">fit</span><span class="p">(</span><span class="n">trainX</span><span class="p">,</span> <span class="n">trainY</span><span class="p">)</span>
    <span class="k">else</span><span class="p">:</span>
        <span class="n">fm</span><span class="o">.</span><span class="n">fit</span><span class="p">(</span><span class="n">trainX</span><span class="p">,</span> <span class="n">trainY</span> <span class="o">*</span> <span class="mf">1.</span><span class="p">)</span>
    <span class="k">return</span> <span class="n">fm</span><span class="o">.</span><span class="n">predict</span><span class="p">(</span><span class="n">testX</span><span class="p">)</span>
</pre></div>

</div>
</div>
</div>

</div>
<div class="cell border-box-sizing text_cell rendered">
<div class="prompt input_prompt">
</div>
<div class="inner_cell">
<div class="text_cell_render border-box-sizing rendered_html">
<h3 id="Executing-all-of-the-tests-takes-much-time">Executing all of the tests takes much time<a class="anchor-link" href="#Executing-all-of-the-tests-takes-much-time">&#182;</a></h3><p>Below is simple mechanism, which preserves results between runs.</p>

</div>
</div>
</div>
<div class="cell border-box-sizing code_cell rendered">
<div class="input">
<div class="prompt input_prompt">In&nbsp;[4]:</div>
<div class="inner_cell">
    <div class="input_area">
<div class=" highlight hl-ipython2"><pre><span class="kn">from</span> <span class="nn">collections</span> <span class="kn">import</span> <span class="n">OrderedDict</span>
<span class="kn">import</span> <span class="nn">time</span>

<span class="n">all_results</span> <span class="o">=</span> <span class="n">OrderedDict</span><span class="p">()</span>
<span class="k">try</span><span class="p">:</span>
    <span class="k">with</span> <span class="nb">open</span><span class="p">(</span><span class="s">&#39;./saved_results.pkl&#39;</span><span class="p">)</span> <span class="k">as</span> <span class="n">f</span><span class="p">:</span>
        <span class="n">all_results</span> <span class="o">=</span> <span class="n">pickle</span><span class="o">.</span><span class="n">load</span><span class="p">(</span><span class="n">f</span><span class="p">)</span>
<span class="k">except</span><span class="p">:</span>
    <span class="k">pass</span>

<span class="k">def</span> <span class="nf">test_on_dataset</span><span class="p">(</span><span class="n">trainX</span><span class="p">,</span> <span class="n">testX</span><span class="p">,</span> <span class="n">trainY</span><span class="p">,</span> <span class="n">testY</span><span class="p">,</span> <span class="n">task_name</span><span class="p">,</span> <span class="n">classification</span><span class="o">=</span><span class="bp">True</span><span class="p">,</span> <span class="n">use_pylibfm</span><span class="o">=</span><span class="bp">True</span><span class="p">):</span>
    <span class="n">algorithms</span> <span class="o">=</span> <span class="n">OrderedDict</span><span class="p">()</span>
    <span class="n">algorithms</span><span class="p">[</span><span class="s">&#39;logistic&#39;</span><span class="p">]</span> <span class="o">=</span> <span class="n">fitpredict_logistic</span>
    <span class="n">algorithms</span><span class="p">[</span><span class="s">&#39;libFM&#39;</span><span class="p">]</span>    <span class="o">=</span> <span class="n">fitpredict_libfm</span>
    <span class="n">algorithms</span><span class="p">[</span><span class="s">&#39;fastFM&#39;</span><span class="p">]</span>   <span class="o">=</span> <span class="n">fitpredict_fastfm</span>
    <span class="k">if</span> <span class="n">use_pylibfm</span><span class="p">:</span>
        <span class="n">algorithms</span><span class="p">[</span><span class="s">&#39;pylibfm&#39;</span><span class="p">]</span>  <span class="o">=</span> <span class="n">fitpredict_pylibfm</span>
    
    <span class="n">results</span> <span class="o">=</span> <span class="n">pandas</span><span class="o">.</span><span class="n">DataFrame</span><span class="p">()</span>
    <span class="k">for</span> <span class="n">name</span><span class="p">,</span> <span class="n">fit_predict</span> <span class="ow">in</span> <span class="n">algorithms</span><span class="o">.</span><span class="n">items</span><span class="p">():</span>
        <span class="n">start</span> <span class="o">=</span> <span class="n">time</span><span class="o">.</span><span class="n">time</span><span class="p">()</span>
        <span class="n">predictions</span> <span class="o">=</span> <span class="n">fit_predict</span><span class="p">(</span><span class="n">trainX</span><span class="p">,</span> <span class="n">trainY</span><span class="p">,</span> <span class="n">testX</span><span class="p">,</span> <span class="n">classification</span><span class="o">=</span><span class="n">classification</span><span class="p">)</span>
        <span class="n">spent_time</span> <span class="o">=</span> <span class="n">time</span><span class="o">.</span><span class="n">time</span><span class="p">()</span> <span class="o">-</span> <span class="n">start</span>
        <span class="n">results</span><span class="o">.</span><span class="n">ix</span><span class="p">[</span><span class="n">name</span><span class="p">,</span> <span class="s">&#39;time&#39;</span><span class="p">]</span> <span class="o">=</span> <span class="n">spent_time</span>
        <span class="k">if</span> <span class="n">classification</span><span class="p">:</span>
            <span class="n">results</span><span class="o">.</span><span class="n">ix</span><span class="p">[</span><span class="n">name</span><span class="p">,</span> <span class="s">&#39;ROC AUC&#39;</span><span class="p">]</span> <span class="o">=</span> <span class="n">roc_auc_score</span><span class="p">(</span><span class="n">testY</span><span class="p">,</span> <span class="n">predictions</span><span class="p">)</span>
        <span class="k">else</span><span class="p">:</span>
            <span class="n">results</span><span class="o">.</span><span class="n">ix</span><span class="p">[</span><span class="n">name</span><span class="p">,</span> <span class="s">&#39;RMSE&#39;</span><span class="p">]</span> <span class="o">=</span> <span class="n">numpy</span><span class="o">.</span><span class="n">mean</span><span class="p">((</span><span class="n">testY</span> <span class="o">-</span> <span class="n">predictions</span><span class="p">)</span> <span class="o">**</span> <span class="mi">2</span><span class="p">)</span> <span class="o">**</span> <span class="mf">0.5</span>
            
    <span class="n">all_results</span><span class="p">[</span><span class="n">task_name</span><span class="p">]</span> <span class="o">=</span> <span class="n">results</span>
    <span class="k">with</span> <span class="nb">open</span><span class="p">(</span><span class="s">&#39;saved_results.pkl&#39;</span><span class="p">,</span> <span class="s">&#39;w&#39;</span><span class="p">)</span> <span class="k">as</span> <span class="n">f</span><span class="p">:</span>
        <span class="n">pickle</span><span class="o">.</span><span class="n">dump</span><span class="p">(</span><span class="n">all_results</span><span class="p">,</span> <span class="n">f</span><span class="p">)</span>
        
    <span class="k">return</span> <span class="n">results</span>
</pre></div>

</div>
</div>
</div>

</div>
<div class="cell border-box-sizing text_cell rendered">
<div class="prompt input_prompt">
</div>
<div class="inner_cell">
<div class="text_cell_render border-box-sizing rendered_html">
<h2 id="Testing-on-movielens-100k-dataset,-only-ids">Testing on movielens-100k dataset, only ids<a class="anchor-link" href="#Testing-on-movielens-100k-dataset,-only-ids">&#182;</a></h2><p>MovieLens dataset is famous dataset in recommender systems. The task is to predict ratings for movies</p>

</div>
</div>
</div>
<div class="cell border-box-sizing code_cell rendered">
<div class="input">
<div class="prompt input_prompt">In&nbsp;[5]:</div>
<div class="inner_cell">
    <div class="input_area">
<div class=" highlight hl-ipython2"><pre><span class="n">trainX</span><span class="p">,</span> <span class="n">testX</span><span class="p">,</span> <span class="n">trainY</span><span class="p">,</span> <span class="n">testY</span> <span class="o">=</span> <span class="n">load_problems</span><span class="o">.</span><span class="n">load_problem_movielens_100k</span><span class="p">(</span><span class="n">all_features</span><span class="o">=</span><span class="bp">False</span><span class="p">)</span>
<span class="n">trainX</span><span class="o">.</span><span class="n">head</span><span class="p">()</span>
</pre></div>

</div>
</div>
</div>

<div class="output_wrapper">
<div class="output">


<div class="output_area"><div class="prompt output_prompt">Out[5]:</div>

<div class="output_html rendered_html output_subarea output_execute_result">
<div>
<table border="1" class="dataframe">
  <thead>
    <tr style="text-align: right;">
      <th></th>
      <th>user</th>
      <th>movie</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <th>98980</th>
      <td>810</td>
      <td>900</td>
    </tr>
    <tr>
      <th>69824</th>
      <td>803</td>
      <td>754</td>
    </tr>
    <tr>
      <th>9928</th>
      <td>51</td>
      <td>286</td>
    </tr>
    <tr>
      <th>75599</th>
      <td>734</td>
      <td>180</td>
    </tr>
    <tr>
      <th>95621</th>
      <td>896</td>
      <td>95</td>
    </tr>
  </tbody>
</table>
</div>
</div>

</div>

</div>
</div>

</div>
<div class="cell border-box-sizing code_cell rendered">
<div class="input">
<div class="prompt input_prompt">In&nbsp;[6]:</div>
<div class="inner_cell">
    <div class="input_area">
<div class=" highlight hl-ipython2"><pre><span class="n">test_on_dataset</span><span class="p">(</span><span class="n">trainX</span><span class="p">,</span> <span class="n">testX</span><span class="p">,</span> <span class="n">trainY</span><span class="p">,</span> <span class="n">testY</span><span class="p">,</span> <span class="n">task_name</span><span class="o">=</span><span class="s">&#39;ml100k, ids&#39;</span><span class="p">,</span> <span class="n">classification</span><span class="o">=</span><span class="bp">False</span><span class="p">)</span>
</pre></div>

</div>
</div>
</div>

<div class="output_wrapper">
<div class="output">


<div class="output_area"><div class="prompt output_prompt">Out[6]:</div>

<div class="output_html rendered_html output_subarea output_execute_result">
<div>
<table border="1" class="dataframe">
  <thead>
    <tr style="text-align: right;">
      <th></th>
      <th>time</th>
      <th>RMSE</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <th>logistic</th>
      <td>0.059469</td>
      <td>0.942771</td>
    </tr>
    <tr>
      <th>libFM</th>
      <td>8.970990</td>
      <td>0.913520</td>
    </tr>
    <tr>
      <th>fastFM</th>
      <td>4.840041</td>
      <td>0.915184</td>
    </tr>
    <tr>
      <th>pylibfm</th>
      <td>13.157164</td>
      <td>0.944870</td>
    </tr>
  </tbody>
</table>
</div>
</div>

</div>

</div>
</div>

</div>
<div class="cell border-box-sizing text_cell rendered">
<div class="prompt input_prompt">
</div>
<div class="inner_cell">
<div class="text_cell_render border-box-sizing rendered_html">
<h2 id="Testing-on-movielens-100k-dataset,-with-additional-information">Testing on movielens-100k dataset, with additional information<a class="anchor-link" href="#Testing-on-movielens-100k-dataset,-with-additional-information">&#182;</a></h2>
</div>
</div>
</div>
<div class="cell border-box-sizing code_cell rendered">
<div class="input">
<div class="prompt input_prompt">In&nbsp;[7]:</div>
<div class="inner_cell">
    <div class="input_area">
<div class=" highlight hl-ipython2"><pre><span class="n">trainX</span><span class="p">,</span> <span class="n">testX</span><span class="p">,</span> <span class="n">trainY</span><span class="p">,</span> <span class="n">testY</span> <span class="o">=</span> <span class="n">load_problems</span><span class="o">.</span><span class="n">load_problem_movielens_100k</span><span class="p">(</span><span class="n">all_features</span><span class="o">=</span><span class="bp">True</span><span class="p">)</span>
<span class="n">trainX</span><span class="o">.</span><span class="n">head</span><span class="p">()</span>
</pre></div>

</div>
</div>
</div>

<div class="output_wrapper">
<div class="output">


<div class="output_area"><div class="prompt output_prompt">Out[7]:</div>

<div class="output_html rendered_html output_subarea output_execute_result">
<div>
<table border="1" class="dataframe">
  <thead>
    <tr style="text-align: right;">
      <th></th>
      <th>user</th>
      <th>movie</th>
      <th>age</th>
      <th>gender</th>
      <th>occupation</th>
      <th>zip</th>
      <th>released</th>
      <th>unknown</th>
      <th>Action</th>
      <th>Adventure</th>
      <th>...</th>
      <th>Fantasy</th>
      <th>Film-Noir</th>
      <th>Horror</th>
      <th>Musical</th>
      <th>Mystery</th>
      <th>Romance</th>
      <th>Sci-Fi</th>
      <th>Thriller</th>
      <th>War</th>
      <th>Western</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <th>98980</th>
      <td>692</td>
      <td>1310</td>
      <td>33</td>
      <td>0</td>
      <td>7</td>
      <td>615</td>
      <td>68</td>
      <td>0</td>
      <td>0</td>
      <td>0</td>
      <td>...</td>
      <td>0</td>
      <td>0</td>
      <td>0</td>
      <td>0</td>
      <td>0</td>
      <td>0</td>
      <td>0</td>
      <td>0</td>
      <td>0</td>
      <td>0</td>
    </tr>
    <tr>
      <th>69824</th>
      <td>931</td>
      <td>528</td>
      <td>48</td>
      <td>1</td>
      <td>3</td>
      <td>59</td>
      <td>57</td>
      <td>0</td>
      <td>0</td>
      <td>0</td>
      <td>...</td>
      <td>0</td>
      <td>0</td>
      <td>0</td>
      <td>0</td>
      <td>0</td>
      <td>0</td>
      <td>0</td>
      <td>0</td>
      <td>0</td>
      <td>0</td>
    </tr>
    <tr>
      <th>9928</th>
      <td>216</td>
      <td>553</td>
      <td>12</td>
      <td>1</td>
      <td>13</td>
      <td>110</td>
      <td>67</td>
      <td>0</td>
      <td>1</td>
      <td>1</td>
      <td>...</td>
      <td>0</td>
      <td>0</td>
      <td>0</td>
      <td>0</td>
      <td>0</td>
      <td>0</td>
      <td>0</td>
      <td>0</td>
      <td>0</td>
      <td>0</td>
    </tr>
    <tr>
      <th>75599</th>
      <td>798</td>
      <td>498</td>
      <td>39</td>
      <td>0</td>
      <td>0</td>
      <td>166</td>
      <td>30</td>
      <td>0</td>
      <td>0</td>
      <td>0</td>
      <td>...</td>
      <td>0</td>
      <td>0</td>
      <td>0</td>
      <td>0</td>
      <td>0</td>
      <td>0</td>
      <td>0</td>
      <td>0</td>
      <td>0</td>
      <td>0</td>
    </tr>
    <tr>
      <th>95621</th>
      <td>910</td>
      <td>547</td>
      <td>27</td>
      <td>0</td>
      <td>20</td>
      <td>397</td>
      <td>68</td>
      <td>0</td>
      <td>0</td>
      <td>0</td>
      <td>...</td>
      <td>1</td>
      <td>0</td>
      <td>0</td>
      <td>0</td>
      <td>0</td>
      <td>0</td>
      <td>0</td>
      <td>0</td>
      <td>0</td>
      <td>0</td>
    </tr>
  </tbody>
</table>
<p>5 rows × 26 columns</p>
</div>
</div>

</div>

</div>
</div>

</div>
<div class="cell border-box-sizing code_cell rendered">
<div class="input">
<div class="prompt input_prompt">In&nbsp;[8]:</div>
<div class="inner_cell">
    <div class="input_area">
<div class=" highlight hl-ipython2"><pre><span class="n">test_on_dataset</span><span class="p">(</span><span class="n">trainX</span><span class="p">,</span> <span class="n">testX</span><span class="p">,</span> <span class="n">trainY</span><span class="p">,</span> <span class="n">testY</span><span class="p">,</span> <span class="n">task_name</span><span class="o">=</span><span class="s">&#39;ml100k&#39;</span><span class="p">,</span> <span class="n">classification</span><span class="o">=</span><span class="bp">False</span><span class="p">)</span>
</pre></div>

</div>
</div>
</div>

<div class="output_wrapper">
<div class="output">


<div class="output_area"><div class="prompt output_prompt">Out[8]:</div>

<div class="output_html rendered_html output_subarea output_execute_result">
<div>
<table border="1" class="dataframe">
  <thead>
    <tr style="text-align: right;">
      <th></th>
      <th>time</th>
      <th>RMSE</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <th>logistic</th>
      <td>1.869114</td>
      <td>0.942377</td>
    </tr>
    <tr>
      <th>libFM</th>
      <td>49.632649</td>
      <td>0.896349</td>
    </tr>
    <tr>
      <th>fastFM</th>
      <td>53.611804</td>
      <td>0.896543</td>
    </tr>
    <tr>
      <th>pylibfm</th>
      <td>55.756278</td>
      <td>NaN</td>
    </tr>
  </tbody>
</table>
</div>
</div>

</div>

</div>
</div>

</div>
<div class="cell border-box-sizing text_cell rendered">
<div class="prompt input_prompt">
</div>
<div class="inner_cell">
<div class="text_cell_render border-box-sizing rendered_html">
<h2 id="Testing-on-movielens-1m-dataset,-only-ids">Testing on movielens-1m dataset, only ids<a class="anchor-link" href="#Testing-on-movielens-1m-dataset,-only-ids">&#182;</a></h2>
</div>
</div>
</div>
<div class="cell border-box-sizing code_cell rendered">
<div class="input">
<div class="prompt input_prompt">In&nbsp;[14]:</div>
<div class="inner_cell">
    <div class="input_area">
<div class=" highlight hl-ipython2"><pre><span class="n">trainX</span><span class="p">,</span> <span class="n">testX</span><span class="p">,</span> <span class="n">trainY</span><span class="p">,</span> <span class="n">testY</span> <span class="o">=</span> <span class="n">load_problems</span><span class="o">.</span><span class="n">load_problem_movielens_1m</span><span class="p">(</span><span class="n">all_features</span><span class="o">=</span><span class="bp">False</span><span class="p">)</span>
<span class="n">trainX</span><span class="o">.</span><span class="n">head</span><span class="p">()</span>
</pre></div>

</div>
</div>
</div>

<div class="output_wrapper">
<div class="output">


<div class="output_area"><div class="prompt"></div>
<div class="output_subarea output_stream output_stderr output_text">
<pre>load_problems.py:73: ParserWarning: Falling back to the &apos;python&apos; engine because the &apos;c&apos; engine does not support regex separators; you can avoid this warning by specifying engine=&apos;python&apos;.
  names=[&apos;user&apos;, &apos;movie&apos;, &apos;rating&apos;, &apos;timestamp&apos;], header=None)
</pre>
</div>
</div>

<div class="output_area"><div class="prompt output_prompt">Out[14]:</div>

<div class="output_html rendered_html output_subarea output_execute_result">
<div>
<table border="1" class="dataframe">
  <thead>
    <tr style="text-align: right;">
      <th></th>
      <th>user</th>
      <th>movie</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <th>610738</th>
      <td>3703</td>
      <td>3541</td>
    </tr>
    <tr>
      <th>324752</th>
      <td>1923</td>
      <td>756</td>
    </tr>
    <tr>
      <th>808217</th>
      <td>4836</td>
      <td>1288</td>
    </tr>
    <tr>
      <th>133807</th>
      <td>866</td>
      <td>1106</td>
    </tr>
    <tr>
      <th>431857</th>
      <td>2630</td>
      <td>2857</td>
    </tr>
  </tbody>
</table>
</div>
</div>

</div>

</div>
</div>

</div>
<div class="cell border-box-sizing code_cell rendered">
<div class="input">
<div class="prompt input_prompt">In&nbsp;[15]:</div>
<div class="inner_cell">
    <div class="input_area">
<div class=" highlight hl-ipython2"><pre><span class="n">test_on_dataset</span><span class="p">(</span><span class="n">trainX</span><span class="p">,</span> <span class="n">testX</span><span class="p">,</span> <span class="n">trainY</span><span class="p">,</span> <span class="n">testY</span><span class="p">,</span> <span class="n">task_name</span><span class="o">=</span><span class="s">&#39;ml-1m,ids&#39;</span><span class="p">,</span> <span class="n">classification</span><span class="o">=</span><span class="bp">False</span><span class="p">)</span>
</pre></div>

</div>
</div>
</div>

<div class="output_wrapper">
<div class="output">


<div class="output_area"><div class="prompt output_prompt">Out[15]:</div>

<div class="output_html rendered_html output_subarea output_execute_result">
<div>
<table border="1" class="dataframe">
  <thead>
    <tr style="text-align: right;">
      <th></th>
      <th>time</th>
      <th>RMSE</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <th>logistic</th>
      <td>1.111601</td>
      <td>0.910718</td>
    </tr>
    <tr>
      <th>libFM</th>
      <td>275.672684</td>
      <td>0.861539</td>
    </tr>
    <tr>
      <th>fastFM</th>
      <td>307.400295</td>
      <td>0.858305</td>
    </tr>
    <tr>
      <th>pylibfm</th>
      <td>132.618739</td>
      <td>0.870263</td>
    </tr>
  </tbody>
</table>
</div>
</div>

</div>

</div>
</div>

</div>
<div class="cell border-box-sizing text_cell rendered">
<div class="prompt input_prompt">
</div>
<div class="inner_cell">
<div class="text_cell_render border-box-sizing rendered_html">
<h2 id="Testing-on-movielens-1m-dataset,-with-additional-information">Testing on movielens-1m dataset, with additional information<a class="anchor-link" href="#Testing-on-movielens-1m-dataset,-with-additional-information">&#182;</a></h2>
</div>
</div>
</div>
<div class="cell border-box-sizing code_cell rendered">
<div class="input">
<div class="prompt input_prompt">In&nbsp;[&nbsp;]:</div>
<div class="inner_cell">
    <div class="input_area">
<div class=" highlight hl-ipython2"><pre><span class="n">trainX</span><span class="p">,</span> <span class="n">testX</span><span class="p">,</span> <span class="n">trainY</span><span class="p">,</span> <span class="n">testY</span> <span class="o">=</span> <span class="n">load_problems</span><span class="o">.</span><span class="n">load_problem_movielens_1m</span><span class="p">(</span><span class="n">all_features</span><span class="o">=</span><span class="bp">True</span><span class="p">)</span>
<span class="n">trainX</span><span class="o">.</span><span class="n">head</span><span class="p">()</span>
</pre></div>

</div>
</div>
</div>

<div class="output_wrapper">
<div class="output">


<div class="output_area"><div class="prompt"></div>
<div class="output_subarea output_stream output_stderr output_text">
<pre>load_problems.py:78: ParserWarning: Falling back to the &apos;python&apos; engine because the &apos;c&apos; engine does not support regex separators; you can avoid this warning by specifying engine=&apos;python&apos;.
  names=[&apos;user&apos;, &apos;gender&apos;, &apos;age&apos;, &apos;occupation&apos;, &apos;zip&apos;], header=None)
load_problems.py:79: ParserWarning: Falling back to the &apos;python&apos; engine because the &apos;c&apos; engine does not support regex separators; you can avoid this warning by specifying engine=&apos;python&apos;.
  movies = pandas.read_csv(folder + &apos;/movies.dat&apos;, sep=&apos;::&apos;, names=[&apos;movie&apos;, &apos;title&apos;, &apos;genres&apos;], header=None)
</pre>
</div>
</div>

<div class="output_area"><div class="prompt output_prompt">Out[&nbsp;]:</div>

<div class="output_html rendered_html output_subarea output_execute_result">
<div>
<table border="1" class="dataframe">
  <thead>
    <tr style="text-align: right;">
      <th></th>
      <th>user</th>
      <th>movie</th>
      <th>gender</th>
      <th>age</th>
      <th>occupation</th>
      <th>zip</th>
      <th>0</th>
      <th>1</th>
      <th>2</th>
      <th>3</th>
      <th>...</th>
      <th>10</th>
      <th>11</th>
      <th>12</th>
      <th>13</th>
      <th>14</th>
      <th>15</th>
      <th>16</th>
      <th>17</th>
      <th>18</th>
      <th>19</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <th>610738</th>
      <td>5245</td>
      <td>2240</td>
      <td>0</td>
      <td>1</td>
      <td>0</td>
      <td>2168</td>
      <td>0</td>
      <td>0</td>
      <td>0</td>
      <td>0</td>
      <td>...</td>
      <td>0</td>
      <td>0</td>
      <td>0</td>
      <td>0</td>
      <td>0</td>
      <td>0</td>
      <td>0</td>
      <td>0</td>
      <td>0</td>
      <td>0</td>
    </tr>
    <tr>
      <th>324752</th>
      <td>4260</td>
      <td>2213</td>
      <td>0</td>
      <td>3</td>
      <td>6</td>
      <td>346</td>
      <td>1</td>
      <td>1</td>
      <td>0</td>
      <td>0</td>
      <td>...</td>
      <td>0</td>
      <td>0</td>
      <td>0</td>
      <td>0</td>
      <td>0</td>
      <td>1</td>
      <td>0</td>
      <td>0</td>
      <td>0</td>
      <td>0</td>
    </tr>
    <tr>
      <th>808217</th>
      <td>481</td>
      <td>1205</td>
      <td>1</td>
      <td>2</td>
      <td>14</td>
      <td>1878</td>
      <td>0</td>
      <td>0</td>
      <td>0</td>
      <td>0</td>
      <td>...</td>
      <td>0</td>
      <td>0</td>
      <td>0</td>
      <td>0</td>
      <td>0</td>
      <td>0</td>
      <td>0</td>
      <td>0</td>
      <td>0</td>
      <td>0</td>
    </tr>
    <tr>
      <th>133807</th>
      <td>973</td>
      <td>1108</td>
      <td>1</td>
      <td>3</td>
      <td>19</td>
      <td>3140</td>
      <td>1</td>
      <td>1</td>
      <td>0</td>
      <td>0</td>
      <td>...</td>
      <td>0</td>
      <td>0</td>
      <td>0</td>
      <td>0</td>
      <td>0</td>
      <td>0</td>
      <td>0</td>
      <td>0</td>
      <td>0</td>
      <td>0</td>
    </tr>
    <tr>
      <th>431857</th>
      <td>1344</td>
      <td>2302</td>
      <td>0</td>
      <td>2</td>
      <td>2</td>
      <td>2336</td>
      <td>0</td>
      <td>0</td>
      <td>0</td>
      <td>0</td>
      <td>...</td>
      <td>0</td>
      <td>0</td>
      <td>0</td>
      <td>0</td>
      <td>0</td>
      <td>1</td>
      <td>0</td>
      <td>0</td>
      <td>0</td>
      <td>0</td>
    </tr>
  </tbody>
</table>
<p>5 rows × 26 columns</p>
</div>
</div>

</div>

</div>
</div>

</div>
<div class="cell border-box-sizing code_cell rendered">
<div class="input">
<div class="prompt input_prompt">In&nbsp;[34]:</div>
<div class="inner_cell">
    <div class="input_area">
<div class=" highlight hl-ipython2"><pre><span class="n">test_on_dataset</span><span class="p">(</span><span class="n">trainX</span><span class="p">,</span> <span class="n">testX</span><span class="p">,</span> <span class="n">trainY</span><span class="p">,</span> <span class="n">testY</span><span class="p">,</span> <span class="n">task_name</span><span class="o">=</span><span class="s">&#39;ml-1m&#39;</span><span class="p">,</span> <span class="n">classification</span><span class="o">=</span><span class="bp">False</span><span class="p">)</span>
</pre></div>

</div>
</div>
</div>

<div class="output_wrapper">
<div class="output">


<div class="output_area"><div class="prompt output_prompt">Out[34]:</div>

<div class="output_html rendered_html output_subarea output_execute_result">
<div>
<table border="1" class="dataframe">
  <thead>
    <tr style="text-align: right;">
      <th></th>
      <th>time</th>
      <th>RMSE</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <th>logistic</th>
      <td>23.983249</td>
      <td>0.911024</td>
    </tr>
    <tr>
      <th>libFM</th>
      <td>779.900802</td>
      <td>0.850382</td>
    </tr>
    <tr>
      <th>fastFM</th>
      <td>1170.468130</td>
      <td>0.852738</td>
    </tr>
    <tr>
      <th>pylibfm</th>
      <td>564.922632</td>
      <td>NaN</td>
    </tr>
  </tbody>
</table>
</div>
</div>

</div>

</div>
</div>

</div>
<div class="cell border-box-sizing text_cell rendered">
<div class="prompt input_prompt">
</div>
<div class="inner_cell">
<div class="text_cell_render border-box-sizing rendered_html">
<h2 id="Test-on-flights-dataset---1m">Test on flights dataset - 1m<a class="anchor-link" href="#Test-on-flights-dataset---1m">&#182;</a></h2><p>Flights dataset is quite famous due to <a href="github.com/szilard/benchm-ml">these benchmarks</a> by szilard.</p>
<p>Based on defferent charateristics the goal is to predict whether the flight was delayed by 15 minutes or more.</p>

</div>
</div>
</div>
<div class="cell border-box-sizing code_cell rendered">
<div class="input">
<div class="prompt input_prompt">In&nbsp;[17]:</div>
<div class="inner_cell">
    <div class="input_area">
<div class=" highlight hl-ipython2"><pre><span class="n">trainX</span><span class="p">,</span> <span class="n">testX</span><span class="p">,</span> <span class="n">trainY</span><span class="p">,</span> <span class="n">testY</span> <span class="o">=</span> <span class="n">load_problems</span><span class="o">.</span><span class="n">load_problem_flight</span><span class="p">(</span><span class="n">large</span><span class="o">=</span><span class="bp">False</span><span class="p">,</span> <span class="n">convert_to_ints</span><span class="o">=</span><span class="bp">False</span><span class="p">)</span>
<span class="n">trainX</span><span class="o">.</span><span class="n">head</span><span class="p">()</span>
</pre></div>

</div>
</div>
</div>

<div class="output_wrapper">
<div class="output">


<div class="output_area"><div class="prompt output_prompt">Out[17]:</div>

<div class="output_html rendered_html output_subarea output_execute_result">
<div>
<table border="1" class="dataframe">
  <thead>
    <tr style="text-align: right;">
      <th></th>
      <th>Month</th>
      <th>DayofMonth</th>
      <th>DayOfWeek</th>
      <th>DepTime</th>
      <th>UniqueCarrier</th>
      <th>Origin</th>
      <th>Dest</th>
      <th>Distance</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <th>0</th>
      <td>c-4</td>
      <td>c-26</td>
      <td>c-2</td>
      <td>1828</td>
      <td>XE</td>
      <td>LEX</td>
      <td>IAH</td>
      <td>828</td>
    </tr>
    <tr>
      <th>1</th>
      <td>c-12</td>
      <td>c-11</td>
      <td>c-1</td>
      <td>1212</td>
      <td>UA</td>
      <td>DEN</td>
      <td>MCI</td>
      <td>533</td>
    </tr>
    <tr>
      <th>2</th>
      <td>c-10</td>
      <td>c-1</td>
      <td>c-6</td>
      <td>935</td>
      <td>OH</td>
      <td>HSV</td>
      <td>CVG</td>
      <td>325</td>
    </tr>
    <tr>
      <th>3</th>
      <td>c-11</td>
      <td>c-26</td>
      <td>c-6</td>
      <td>930</td>
      <td>OH</td>
      <td>JFK</td>
      <td>PNS</td>
      <td>1028</td>
    </tr>
    <tr>
      <th>4</th>
      <td>c-12</td>
      <td>c-6</td>
      <td>c-2</td>
      <td>1350</td>
      <td>MQ</td>
      <td>DFW</td>
      <td>LBB</td>
      <td>282</td>
    </tr>
  </tbody>
</table>
</div>
</div>

</div>

</div>
</div>

</div>
<div class="cell border-box-sizing code_cell rendered">
<div class="input">
<div class="prompt input_prompt">In&nbsp;[24]:</div>
<div class="inner_cell">
    <div class="input_area">
<div class=" highlight hl-ipython2"><pre><span class="n">trainX</span><span class="p">,</span> <span class="n">testX</span><span class="p">,</span> <span class="n">trainY</span><span class="p">,</span> <span class="n">testY</span> <span class="o">=</span> <span class="n">load_problems</span><span class="o">.</span><span class="n">load_problem_flight</span><span class="p">(</span><span class="n">large</span><span class="o">=</span><span class="bp">False</span><span class="p">,</span> <span class="n">convert_to_ints</span><span class="o">=</span><span class="bp">True</span><span class="p">)</span>
<span class="n">trainX</span><span class="o">.</span><span class="n">head</span><span class="p">()</span>
</pre></div>

</div>
</div>
</div>

<div class="output_wrapper">
<div class="output">


<div class="output_area"><div class="prompt output_prompt">Out[24]:</div>

<div class="output_html rendered_html output_subarea output_execute_result">
<div>
<table border="1" class="dataframe">
  <thead>
    <tr style="text-align: right;">
      <th></th>
      <th>Month</th>
      <th>DayofMonth</th>
      <th>DayOfWeek</th>
      <th>DepTime</th>
      <th>UniqueCarrier</th>
      <th>Origin</th>
      <th>Dest</th>
      <th>Distance</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <th>0</th>
      <td>7</td>
      <td>19</td>
      <td>2</td>
      <td>8</td>
      <td>21</td>
      <td>157</td>
      <td>133</td>
      <td>6</td>
    </tr>
    <tr>
      <th>1</th>
      <td>4</td>
      <td>3</td>
      <td>1</td>
      <td>4</td>
      <td>18</td>
      <td>79</td>
      <td>172</td>
      <td>4</td>
    </tr>
    <tr>
      <th>2</th>
      <td>2</td>
      <td>1</td>
      <td>6</td>
      <td>2</td>
      <td>15</td>
      <td>129</td>
      <td>72</td>
      <td>2</td>
    </tr>
    <tr>
      <th>3</th>
      <td>3</td>
      <td>19</td>
      <td>6</td>
      <td>2</td>
      <td>15</td>
      <td>147</td>
      <td>220</td>
      <td>7</td>
    </tr>
    <tr>
      <th>4</th>
      <td>4</td>
      <td>28</td>
      <td>2</td>
      <td>5</td>
      <td>13</td>
      <td>80</td>
      <td>155</td>
      <td>2</td>
    </tr>
  </tbody>
</table>
</div>
</div>

</div>

</div>
</div>

</div>
<div class="cell border-box-sizing code_cell rendered">
<div class="input">
<div class="prompt input_prompt">In&nbsp;[23]:</div>
<div class="inner_cell">
    <div class="input_area">
<div class=" highlight hl-ipython2"><pre><span class="n">test_on_dataset</span><span class="p">(</span><span class="n">trainX</span><span class="p">,</span> <span class="n">testX</span><span class="p">,</span> <span class="n">trainY</span><span class="p">,</span> <span class="n">testY</span><span class="p">,</span> <span class="n">task_name</span><span class="o">=</span><span class="s">&#39;flight1m&#39;</span><span class="p">,</span> <span class="n">classification</span><span class="o">=</span><span class="bp">True</span><span class="p">)</span>
</pre></div>

</div>
</div>
</div>

<div class="output_wrapper">
<div class="output">


<div class="output_area"><div class="prompt output_prompt">Out[23]:</div>

<div class="output_html rendered_html output_subarea output_execute_result">
<div>
<table border="1" class="dataframe">
  <thead>
    <tr style="text-align: right;">
      <th></th>
      <th>time</th>
      <th>ROC AUC</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <th>logistic</th>
      <td>26.408147</td>
      <td>0.715099</td>
    </tr>
    <tr>
      <th>libFM</th>
      <td>441.534214</td>
      <td>0.720484</td>
    </tr>
    <tr>
      <th>fastFM</th>
      <td>667.197644</td>
      <td>0.718840</td>
    </tr>
    <tr>
      <th>pylibfm</th>
      <td>316.149734</td>
      <td>0.711200</td>
    </tr>
  </tbody>
</table>
</div>
</div>

</div>

</div>
</div>

</div>
<div class="cell border-box-sizing text_cell rendered">
<div class="prompt input_prompt">
</div>
<div class="inner_cell">
<div class="text_cell_render border-box-sizing rendered_html">
<h2 id="Test-on-flights-dataset---10m">Test on flights dataset - 10m<a class="anchor-link" href="#Test-on-flights-dataset---10m">&#182;</a></h2><p>pylibFM drops the kernel, so doesn't participate in comparison</p>

</div>
</div>
</div>
<div class="cell border-box-sizing code_cell rendered">
<div class="input">
<div class="prompt input_prompt">In&nbsp;[&nbsp;]:</div>
<div class="inner_cell">
    <div class="input_area">
<div class=" highlight hl-ipython2"><pre><span class="n">trainX</span><span class="p">,</span> <span class="n">testX</span><span class="p">,</span> <span class="n">trainY</span><span class="p">,</span> <span class="n">testY</span> <span class="o">=</span> <span class="n">load_problems</span><span class="o">.</span><span class="n">load_problem_flight</span><span class="p">(</span><span class="n">large</span><span class="o">=</span><span class="bp">True</span><span class="p">,</span> <span class="n">convert_to_ints</span><span class="o">=</span><span class="bp">True</span><span class="p">)</span>
<span class="n">trainX</span><span class="o">.</span><span class="n">head</span><span class="p">()</span>
</pre></div>

</div>
</div>
</div>

<div class="output_wrapper">
<div class="output">


<div class="output_area"><div class="prompt output_prompt">Out[&nbsp;]:</div>

<div class="output_html rendered_html output_subarea output_execute_result">
<div>
<table border="1" class="dataframe">
  <thead>
    <tr style="text-align: right;">
      <th></th>
      <th>Month</th>
      <th>DayofMonth</th>
      <th>DayOfWeek</th>
      <th>DepTime</th>
      <th>UniqueCarrier</th>
      <th>Origin</th>
      <th>Dest</th>
      <th>Distance</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <th>0</th>
      <td>8</td>
      <td>3</td>
      <td>4</td>
      <td>0</td>
      <td>15</td>
      <td>37</td>
      <td>29</td>
      <td>1</td>
    </tr>
    <tr>
      <th>1</th>
      <td>6</td>
      <td>16</td>
      <td>3</td>
      <td>8</td>
      <td>1</td>
      <td>149</td>
      <td>157</td>
      <td>9</td>
    </tr>
    <tr>
      <th>2</th>
      <td>3</td>
      <td>13</td>
      <td>7</td>
      <td>0</td>
      <td>13</td>
      <td>29</td>
      <td>37</td>
      <td>1</td>
    </tr>
    <tr>
      <th>3</th>
      <td>11</td>
      <td>27</td>
      <td>6</td>
      <td>4</td>
      <td>14</td>
      <td>80</td>
      <td>198</td>
      <td>7</td>
    </tr>
    <tr>
      <th>4</th>
      <td>7</td>
      <td>19</td>
      <td>2</td>
      <td>1</td>
      <td>12</td>
      <td>216</td>
      <td>155</td>
      <td>9</td>
    </tr>
  </tbody>
</table>
</div>
</div>

</div>

</div>
</div>

</div>
<div class="cell border-box-sizing code_cell rendered">
<div class="input">
<div class="prompt input_prompt">In&nbsp;[11]:</div>
<div class="inner_cell">
    <div class="input_area">
<div class=" highlight hl-ipython2"><pre><span class="n">test_on_dataset</span><span class="p">(</span><span class="n">trainX</span><span class="p">,</span> <span class="n">testX</span><span class="p">,</span> <span class="n">trainY</span><span class="p">,</span> <span class="n">testY</span><span class="p">,</span> <span class="n">task_name</span><span class="o">=</span><span class="s">&#39;flight10m&#39;</span><span class="p">,</span> <span class="n">classification</span><span class="o">=</span><span class="bp">True</span><span class="p">,</span> <span class="n">use_pylibfm</span><span class="o">=</span><span class="bp">False</span><span class="p">)</span>
</pre></div>

</div>
</div>
</div>

<div class="output_wrapper">
<div class="output">


<div class="output_area"><div class="prompt output_prompt">Out[11]:</div>

<div class="output_html rendered_html output_subarea output_execute_result">
<div>
<table border="1" class="dataframe">
  <thead>
    <tr style="text-align: right;">
      <th></th>
      <th>time</th>
      <th>ROC AUC</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <th>logistic</th>
      <td>307.085924</td>
      <td>0.715754</td>
    </tr>
    <tr>
      <th>libFM</th>
      <td>8500.038059</td>
      <td>0.724258</td>
    </tr>
    <tr>
      <th>fastFM</th>
      <td>10718.261802</td>
      <td>0.721615</td>
    </tr>
  </tbody>
</table>
</div>
</div>

</div>

</div>
</div>

</div>
<div class="cell border-box-sizing text_cell rendered">
<div class="prompt input_prompt">
</div>
<div class="inner_cell">
<div class="text_cell_render border-box-sizing rendered_html">
<h2 id="Flights-dataset-with-additional-features">Flights dataset with additional features<a class="anchor-link" href="#Flights-dataset-with-additional-features">&#182;</a></h2><p>We simply add some 'quadratic' features</p>

</div>
</div>
</div>
<div class="cell border-box-sizing code_cell rendered">
<div class="input">
<div class="prompt input_prompt">In&nbsp;[63]:</div>
<div class="inner_cell">
    <div class="input_area">
<div class=" highlight hl-ipython2"><pre><span class="n">trainX</span><span class="p">,</span> <span class="n">testX</span><span class="p">,</span> <span class="n">trainY</span><span class="p">,</span> <span class="n">testY</span> <span class="o">=</span> <span class="n">load_problems</span><span class="o">.</span><span class="n">load_problem_flight_extended</span><span class="p">(</span><span class="n">large</span><span class="o">=</span><span class="bp">False</span><span class="p">)</span>
<span class="n">trainX</span><span class="o">.</span><span class="n">head</span><span class="p">()</span>
</pre></div>

</div>
</div>
</div>

<div class="output_wrapper">
<div class="output">


<div class="output_area"><div class="prompt output_prompt">Out[63]:</div>

<div class="output_html rendered_html output_subarea output_execute_result">
<div>
<table border="1" class="dataframe">
  <thead>
    <tr style="text-align: right;">
      <th></th>
      <th>Month</th>
      <th>DayofMonth</th>
      <th>DayOfWeek</th>
      <th>DepTime</th>
      <th>UniqueCarrier</th>
      <th>Origin</th>
      <th>Dest</th>
      <th>Distance</th>
      <th>UniqueCarrier_Origin</th>
      <th>UniqueCarrier_Dest</th>
      <th>UniqueCarrier_DepTime</th>
      <th>Origin_Dest</th>
      <th>Origin_DepTime</th>
      <th>Dest_DepTime</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <th>0</th>
      <td>7</td>
      <td>19</td>
      <td>2</td>
      <td>19</td>
      <td>21</td>
      <td>157</td>
      <td>133</td>
      <td>7</td>
      <td>1628</td>
      <td>1622</td>
      <td>496</td>
      <td>2606</td>
      <td>2612</td>
      <td>2275</td>
    </tr>
    <tr>
      <th>1</th>
      <td>4</td>
      <td>3</td>
      <td>1</td>
      <td>13</td>
      <td>18</td>
      <td>79</td>
      <td>172</td>
      <td>5</td>
      <td>1342</td>
      <td>1369</td>
      <td>417</td>
      <td>1267</td>
      <td>1294</td>
      <td>2921</td>
    </tr>
    <tr>
      <th>2</th>
      <td>2</td>
      <td>1</td>
      <td>6</td>
      <td>10</td>
      <td>15</td>
      <td>129</td>
      <td>72</td>
      <td>3</td>
      <td>1072</td>
      <td>1050</td>
      <td>344</td>
      <td>1985</td>
      <td>2117</td>
      <td>1195</td>
    </tr>
    <tr>
      <th>3</th>
      <td>3</td>
      <td>19</td>
      <td>6</td>
      <td>10</td>
      <td>15</td>
      <td>147</td>
      <td>220</td>
      <td>8</td>
      <td>1083</td>
      <td>1111</td>
      <td>344</td>
      <td>2363</td>
      <td>2410</td>
      <td>3807</td>
    </tr>
    <tr>
      <th>4</th>
      <td>4</td>
      <td>28</td>
      <td>2</td>
      <td>14</td>
      <td>13</td>
      <td>80</td>
      <td>155</td>
      <td>3</td>
      <td>806</td>
      <td>836</td>
      <td>299</td>
      <td>1379</td>
      <td>1317</td>
      <td>2665</td>
    </tr>
  </tbody>
</table>
</div>
</div>

</div>

</div>
</div>

</div>
<div class="cell border-box-sizing code_cell rendered">
<div class="input">
<div class="prompt input_prompt">In&nbsp;[64]:</div>
<div class="inner_cell">
    <div class="input_area">
<div class=" highlight hl-ipython2"><pre><span class="n">test_on_dataset</span><span class="p">(</span><span class="n">trainX</span><span class="p">,</span> <span class="n">testX</span><span class="p">,</span> <span class="n">trainY</span><span class="p">,</span> <span class="n">testY</span><span class="p">,</span> <span class="n">task_name</span><span class="o">=</span><span class="s">&#39;flight1m, ext&#39;</span><span class="p">,</span> <span class="n">classification</span><span class="o">=</span><span class="bp">True</span><span class="p">)</span>
</pre></div>

</div>
</div>
</div>

<div class="output_wrapper">
<div class="output">


<div class="output_area"><div class="prompt output_prompt">Out[64]:</div>

<div class="output_html rendered_html output_subarea output_execute_result">
<div>
<table border="1" class="dataframe">
  <thead>
    <tr style="text-align: right;">
      <th></th>
      <th>time</th>
      <th>ROC AUC</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <th>logistic</th>
      <td>130.553217</td>
      <td>0.739113</td>
    </tr>
    <tr>
      <th>libFM</th>
      <td>1284.598730</td>
      <td>0.760524</td>
    </tr>
    <tr>
      <th>fastFM</th>
      <td>2398.107206</td>
      <td>0.748448</td>
    </tr>
    <tr>
      <th>pylibfm</th>
      <td>514.768003</td>
      <td>0.743312</td>
    </tr>
  </tbody>
</table>
</div>
</div>

</div>

</div>
</div>

</div>
<div class="cell border-box-sizing text_cell rendered">
<div class="prompt input_prompt">
</div>
<div class="inner_cell">
<div class="text_cell_render border-box-sizing rendered_html">
<h2 id="Test-on-Avazu-dataset-(100k)">Test on Avazu dataset (100k)<a class="anchor-link" href="#Test-on-Avazu-dataset-(100k)">&#182;</a></h2><p>Avazu dataset comes from kaggle challenge, goal is to predict Click-Through Rate.</p>
<p>All the variables given are categorical, LibFM gave good results in this challenge.</p>

</div>
</div>
</div>
<div class="cell border-box-sizing code_cell rendered">
<div class="input">
<div class="prompt input_prompt">In&nbsp;[9]:</div>
<div class="inner_cell">
    <div class="input_area">
<div class=" highlight hl-ipython2"><pre><span class="n">trainX</span><span class="p">,</span> <span class="n">testX</span><span class="p">,</span> <span class="n">trainY</span><span class="p">,</span> <span class="n">testY</span> <span class="o">=</span> <span class="n">load_problems</span><span class="o">.</span><span class="n">load_problem_ad</span><span class="p">(</span><span class="n">train_size</span><span class="o">=</span><span class="mi">100000</span><span class="p">)</span>
<span class="c"># taking max hash of 1000 for each category</span>
<span class="n">trainX</span> <span class="o">=</span> <span class="n">trainX</span> <span class="o">%</span> <span class="mi">1000</span>
<span class="n">testX</span> <span class="o">=</span> <span class="n">testX</span> <span class="o">%</span> <span class="mi">1000</span>
<span class="n">trainX</span><span class="o">.</span><span class="n">head</span><span class="p">()</span>
</pre></div>

</div>
</div>
</div>

<div class="output_wrapper">
<div class="output">


<div class="output_area"><div class="prompt output_prompt">Out[9]:</div>

<div class="output_html rendered_html output_subarea output_execute_result">
<div>
<table border="1" class="dataframe">
  <thead>
    <tr style="text-align: right;">
      <th></th>
      <th>hour</th>
      <th>C1</th>
      <th>banner_pos</th>
      <th>site_id</th>
      <th>site_domain</th>
      <th>site_category</th>
      <th>app_id</th>
      <th>app_domain</th>
      <th>app_category</th>
      <th>device_id</th>
      <th>...</th>
      <th>device_conn_type</th>
      <th>C14</th>
      <th>C15</th>
      <th>C16</th>
      <th>C17</th>
      <th>C18</th>
      <th>C19</th>
      <th>C20</th>
      <th>C21</th>
      <th>day</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <th>13458308</th>
      <td>1</td>
      <td>2</td>
      <td>0</td>
      <td>582</td>
      <td>339</td>
      <td>2</td>
      <td>884</td>
      <td>254</td>
      <td>0</td>
      <td>800</td>
      <td>...</td>
      <td>0</td>
      <td>281</td>
      <td>3</td>
      <td>2</td>
      <td>56</td>
      <td>0</td>
      <td>2</td>
      <td>0</td>
      <td>22</td>
      <td>24</td>
    </tr>
    <tr>
      <th>11331426</th>
      <td>10</td>
      <td>2</td>
      <td>0</td>
      <td>582</td>
      <td>339</td>
      <td>2</td>
      <td>884</td>
      <td>254</td>
      <td>0</td>
      <td>800</td>
      <td>...</td>
      <td>1</td>
      <td>283</td>
      <td>3</td>
      <td>2</td>
      <td>56</td>
      <td>0</td>
      <td>2</td>
      <td>0</td>
      <td>22</td>
      <td>23</td>
    </tr>
    <tr>
      <th>1271792</th>
      <td>6</td>
      <td>2</td>
      <td>0</td>
      <td>879</td>
      <td>903</td>
      <td>24</td>
      <td>884</td>
      <td>254</td>
      <td>0</td>
      <td>800</td>
      <td>...</td>
      <td>0</td>
      <td>84</td>
      <td>3</td>
      <td>2</td>
      <td>18</td>
      <td>3</td>
      <td>14</td>
      <td>57</td>
      <td>6</td>
      <td>21</td>
    </tr>
    <tr>
      <th>7618971</th>
      <td>13</td>
      <td>2</td>
      <td>0</td>
      <td>762</td>
      <td>841</td>
      <td>24</td>
      <td>884</td>
      <td>254</td>
      <td>0</td>
      <td>800</td>
      <td>...</td>
      <td>0</td>
      <td>88</td>
      <td>3</td>
      <td>2</td>
      <td>220</td>
      <td>0</td>
      <td>2</td>
      <td>132</td>
      <td>42</td>
      <td>22</td>
    </tr>
    <tr>
      <th>16882663</th>
      <td>3</td>
      <td>2</td>
      <td>1</td>
      <td>158</td>
      <td>893</td>
      <td>24</td>
      <td>884</td>
      <td>254</td>
      <td>0</td>
      <td>800</td>
      <td>...</td>
      <td>0</td>
      <td>376</td>
      <td>3</td>
      <td>2</td>
      <td>88</td>
      <td>2</td>
      <td>4</td>
      <td>13</td>
      <td>8</td>
      <td>25</td>
    </tr>
  </tbody>
</table>
<p>5 rows × 23 columns</p>
</div>
</div>

</div>

</div>
</div>

</div>
<div class="cell border-box-sizing code_cell rendered">
<div class="input">
<div class="prompt input_prompt">In&nbsp;[36]:</div>
<div class="inner_cell">
    <div class="input_area">
<div class=" highlight hl-ipython2"><pre><span class="n">test_on_dataset</span><span class="p">(</span><span class="n">trainX</span><span class="p">,</span> <span class="n">testX</span><span class="p">,</span> <span class="n">trainY</span><span class="p">,</span> <span class="n">testY</span><span class="p">,</span> 
                <span class="n">task_name</span><span class="o">=</span><span class="s">&#39;avazu100k&#39;</span><span class="p">,</span> <span class="n">classification</span><span class="o">=</span><span class="bp">True</span><span class="p">,</span> <span class="n">use_pylibfm</span><span class="o">=</span><span class="bp">False</span><span class="p">)</span>
</pre></div>

</div>
</div>
</div>

<div class="output_wrapper">
<div class="output">


<div class="output_area"><div class="prompt output_prompt">Out[36]:</div>

<div class="output_html rendered_html output_subarea output_execute_result">
<div>
<table border="1" class="dataframe">
  <thead>
    <tr style="text-align: right;">
      <th></th>
      <th>time</th>
      <th>ROC AUC</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <th>logistic</th>
      <td>40.417285</td>
      <td>0.717096</td>
    </tr>
    <tr>
      <th>libFM</th>
      <td>7778.470057</td>
      <td>0.730173</td>
    </tr>
    <tr>
      <th>fastFM</th>
      <td>6601.202590</td>
      <td>0.699962</td>
    </tr>
  </tbody>
</table>
</div>
</div>

</div>

</div>
</div>

</div>
<div class="cell border-box-sizing text_cell rendered">
<div class="prompt input_prompt">
</div>
<div class="inner_cell">
<div class="text_cell_render border-box-sizing rendered_html">
<h2 id="Avazu-1m">Avazu 1m<a class="anchor-link" href="#Avazu-1m">&#182;</a></h2>
</div>
</div>
</div>
<div class="cell border-box-sizing code_cell rendered">
<div class="input">
<div class="prompt input_prompt">In&nbsp;[35]:</div>
<div class="inner_cell">
    <div class="input_area">
<div class=" highlight hl-ipython2"><pre><span class="n">trainX</span><span class="p">,</span> <span class="n">testX</span><span class="p">,</span> <span class="n">trainY</span><span class="p">,</span> <span class="n">testY</span> <span class="o">=</span> <span class="n">load_problems</span><span class="o">.</span><span class="n">load_problem_ad</span><span class="p">(</span><span class="n">train_size</span><span class="o">=</span><span class="mi">1000000</span><span class="p">)</span>
<span class="c"># taking max hash of 1000 for each category</span>
<span class="n">trainX</span> <span class="o">=</span> <span class="n">trainX</span> <span class="o">%</span> <span class="mi">1000</span>
<span class="n">testX</span> <span class="o">=</span> <span class="n">testX</span> <span class="o">%</span> <span class="mi">1000</span>
<span class="n">test_on_dataset</span><span class="p">(</span><span class="n">trainX</span><span class="p">,</span> <span class="n">testX</span><span class="p">,</span> <span class="n">trainY</span><span class="p">,</span> <span class="n">testY</span><span class="p">,</span> 
                <span class="n">task_name</span><span class="o">=</span><span class="s">&#39;avazu1m&#39;</span><span class="p">,</span> <span class="n">classification</span><span class="o">=</span><span class="bp">True</span><span class="p">,</span> <span class="n">use_pylibfm</span><span class="o">=</span><span class="bp">False</span><span class="p">)</span>
</pre></div>

</div>
</div>
</div>

<div class="output_wrapper">
<div class="output">


<div class="output_area"><div class="prompt output_prompt">Out[35]:</div>

<div class="output_html rendered_html output_subarea output_execute_result">
<div>
<table border="1" class="dataframe">
  <thead>
    <tr style="text-align: right;">
      <th></th>
      <th>time</th>
      <th>ROC AUC</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <th>logistic</th>
      <td>228.501853</td>
      <td>0.740987</td>
    </tr>
    <tr>
      <th>libFM</th>
      <td>9109.968575</td>
      <td>0.748516</td>
    </tr>
    <tr>
      <th>fastFM</th>
      <td>7962.800504</td>
      <td>0.733875</td>
    </tr>
  </tbody>
</table>
</div>
</div>

</div>

</div>
</div>

</div>
<div class="cell border-box-sizing text_cell rendered">
<div class="prompt input_prompt">
</div>
<div class="inner_cell">
<div class="text_cell_render border-box-sizing rendered_html">
<h1 id="Results">Results<a class="anchor-link" href="#Results">&#182;</a></h1><p>composing all results in one table. 
RMSE should be minimal, ROC AUC - maximal.</p>

</div>
</div>
</div>
<div class="cell border-box-sizing code_cell rendered">
<div class="input">
<div class="prompt input_prompt">In&nbsp;[69]:</div>
<div class="inner_cell">
    <div class="input_area">
<div class=" highlight hl-ipython2"><pre><span class="n">results_table</span> <span class="o">=</span> <span class="n">pandas</span><span class="o">.</span><span class="n">DataFrame</span><span class="p">()</span>
<span class="n">tuples</span> <span class="o">=</span> <span class="p">[]</span>

<span class="k">for</span> <span class="n">name</span> <span class="ow">in</span> <span class="p">[</span><span class="s">&#39;ml100k, ids&#39;</span><span class="p">,</span> <span class="s">&#39;ml-1m,ids&#39;</span><span class="p">,</span> <span class="s">&#39;ml100k&#39;</span><span class="p">,</span> <span class="s">&#39;ml-1m&#39;</span><span class="p">,</span> <span class="s">&#39;flight1m&#39;</span><span class="p">,</span> <span class="s">&#39;flight1m, ext&#39;</span><span class="p">,</span> <span class="s">&#39;flight10m&#39;</span><span class="p">,</span> <span class="s">&#39;avazu100k&#39;</span><span class="p">,</span> <span class="s">&#39;avazu1m&#39;</span><span class="p">]:</span>
    <span class="n">df</span> <span class="o">=</span> <span class="n">all_results</span><span class="p">[</span><span class="n">name</span><span class="p">]</span>
    <span class="n">results_table</span><span class="p">[</span><span class="n">name</span> <span class="o">+</span> <span class="s">&#39; (time)&#39;</span><span class="p">]</span> <span class="o">=</span> <span class="n">df</span><span class="p">[</span><span class="s">&#39;time&#39;</span><span class="p">]</span>
    <span class="n">metric_name</span> <span class="o">=</span> <span class="n">df</span><span class="o">.</span><span class="n">columns</span><span class="p">[</span><span class="o">-</span><span class="mi">1</span><span class="p">]</span>
    <span class="n">results_table</span><span class="p">[</span><span class="n">name</span> <span class="o">+</span> <span class="n">metric_name</span><span class="p">]</span> <span class="o">=</span> <span class="n">df</span><span class="p">[</span><span class="n">metric_name</span><span class="p">]</span>
    <span class="n">tuples</span><span class="o">.</span><span class="n">append</span><span class="p">([</span><span class="n">name</span><span class="p">,</span> <span class="s">&#39;time&#39;</span><span class="p">])</span>
    <span class="n">tuples</span><span class="o">.</span><span class="n">append</span><span class="p">([</span><span class="n">name</span><span class="p">,</span> <span class="n">df</span><span class="o">.</span><span class="n">columns</span><span class="p">[</span><span class="o">-</span><span class="mi">1</span><span class="p">]])</span>
    
<span class="n">results_table</span> <span class="o">=</span> <span class="n">results_table</span><span class="o">.</span><span class="n">T</span>
<span class="n">results_table</span><span class="o">.</span><span class="n">index</span> <span class="o">=</span> <span class="n">pandas</span><span class="o">.</span><span class="n">MultiIndex</span><span class="o">.</span><span class="n">from_tuples</span><span class="p">(</span><span class="n">tuples</span><span class="p">,</span> <span class="n">names</span><span class="o">=</span><span class="p">[</span><span class="s">&#39;dataset&#39;</span><span class="p">,</span> <span class="s">&#39;value&#39;</span><span class="p">])</span>
<span class="n">results_table</span><span class="o">.</span><span class="n">T</span>
</pre></div>

</div>
</div>
</div>

<div class="output_wrapper">
<div class="output">


<div class="output_area"><div class="prompt output_prompt">Out[69]:</div>

<div class="output_html rendered_html output_subarea output_execute_result">
<div>
<table border="1" class="dataframe">
  <thead>
    <tr>
      <th>dataset</th>
      <th colspan="2" halign="left">ml100k, ids</th>
      <th colspan="2" halign="left">ml-1m,ids</th>
      <th colspan="2" halign="left">ml100k</th>
      <th colspan="2" halign="left">ml-1m</th>
      <th colspan="2" halign="left">flight1m</th>
      <th colspan="2" halign="left">flight1m, ext</th>
      <th colspan="2" halign="left">flight10m</th>
      <th colspan="2" halign="left">avazu100k</th>
      <th colspan="2" halign="left">avazu1m</th>
    </tr>
    <tr>
      <th>value</th>
      <th>time</th>
      <th>RMSE</th>
      <th>time</th>
      <th>RMSE</th>
      <th>time</th>
      <th>RMSE</th>
      <th>time</th>
      <th>RMSE</th>
      <th>time</th>
      <th>ROC AUC</th>
      <th>time</th>
      <th>ROC AUC</th>
      <th>time</th>
      <th>ROC AUC</th>
      <th>time</th>
      <th>ROC AUC</th>
      <th>time</th>
      <th>ROC AUC</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <th>logistic</th>
      <td>0.059469</td>
      <td>0.942771</td>
      <td>1.111601</td>
      <td>0.910718</td>
      <td>1.869114</td>
      <td>0.942377</td>
      <td>23.983249</td>
      <td>0.911024</td>
      <td>26.408147</td>
      <td>0.715099</td>
      <td>130.553217</td>
      <td>0.739113</td>
      <td>307.085924</td>
      <td>0.715754</td>
      <td>40.417285</td>
      <td>0.717096</td>
      <td>228.501853</td>
      <td>0.740987</td>
    </tr>
    <tr>
      <th>libFM</th>
      <td>8.970990</td>
      <td>0.913520</td>
      <td>275.672684</td>
      <td>0.861539</td>
      <td>49.632649</td>
      <td>0.896349</td>
      <td>779.900802</td>
      <td>0.850382</td>
      <td>441.534214</td>
      <td>0.720484</td>
      <td>1284.598730</td>
      <td>0.760524</td>
      <td>8500.038059</td>
      <td>0.724258</td>
      <td>7778.470057</td>
      <td>0.730173</td>
      <td>9109.968575</td>
      <td>0.748516</td>
    </tr>
    <tr>
      <th>fastFM</th>
      <td>4.840041</td>
      <td>0.915184</td>
      <td>307.400295</td>
      <td>0.858305</td>
      <td>53.611804</td>
      <td>0.896543</td>
      <td>1170.468130</td>
      <td>0.852738</td>
      <td>667.197644</td>
      <td>0.718840</td>
      <td>2398.107206</td>
      <td>0.748448</td>
      <td>10718.261802</td>
      <td>0.721615</td>
      <td>6601.202590</td>
      <td>0.699962</td>
      <td>7962.800504</td>
      <td>0.733875</td>
    </tr>
    <tr>
      <th>pylibfm</th>
      <td>13.157164</td>
      <td>0.944870</td>
      <td>132.618739</td>
      <td>0.870263</td>
      <td>55.756278</td>
      <td>NaN</td>
      <td>564.922632</td>
      <td>NaN</td>
      <td>316.149734</td>
      <td>0.711200</td>
      <td>514.768003</td>
      <td>0.743312</td>
      <td>NaN</td>
      <td>NaN</td>
      <td>NaN</td>
      <td>NaN</td>
      <td>NaN</td>
      <td>NaN</td>
    </tr>
  </tbody>
</table>
</div>
</div>

</div>

</div>
</div>

</div>
<div class="cell border-box-sizing text_cell rendered">
<div class="prompt input_prompt">
</div>
<div class="inner_cell">
<div class="text_cell_render border-box-sizing rendered_html">
<h1 id="Conclusion">Conclusion<a class="anchor-link" href="#Conclusion">&#182;</a></h1><ul>
<li><code>pylibfm</code> is out of the game. It is slow, it crashes on large datasets, sometimes simply diverge and hardly can compete in quality. <br />
 Nothing new, adaptive methods require babysitting</li>
<li><code>FastFM</code> and <code>LibFM</code> are quite stable and fast</li>
<li>but <code>LibFM</code>, being a bit faster, on average provides much better results.</li>
</ul>
<p>As a sidenote, we saw on example with flight dataset that some feature engineering with providing quadratic features gives very significant boost in quality - even logisitic regression can work much better and faster than FMs on original features.</p>

</div>
</div>
</div>

<p>
    This post was written in IPython. You can download the notebook from
     <a href='https://github.com/arogozhnikov/arogozhnikov.github.io/tree/master/notebooks'>
    repository</a>.
</p>

